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content  is  based  on  the  author's  experience  in  OT&E,  interviews  with  DA  and 
DOD  staff  personnel,  and  research  of  OT&E  policies  and  test  literature 

In  the  report, /the  history  and  evolution  of  the  Army's  current  OT^E  structure 
are  traced.  The  organization  of  the  US  Arav  Operational  and  Test  and  Evaluation 
Agency  is  discussed.  In  an  analysis,  fiv^areas  of  major  Congressional  criti- 
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Army' 8 current  policies  and  procedures  which  ameliorate  the  criticisms  are 
cited.  The  Army's  0T/p2  policies  are  £ounff~to“be  respoftsiVe' to  history” of  the"~^ f\ 
recent  past.  Present  OT^IT~procedures:  provide  adequate  evaluation  of  a new 

materiel  system  prior  to  a production  commitment;  respond  to  user  needs;  assist 
in  measuring  incremental  effectiveness  of  new  systems;  and  produce  an  indepen- 
dent and  objective  evaluation  of  the  operational  effectiveness  of  new  materiel. 
Home  future  areas  of  Interest  are  discussed  which  will  assist  the  continuation 


of  a viable  OT^E  organization  within  the  Army. 
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EXECUTIVE  SUMMARY 


Starting  In  the  1960's,  the  testing  community  of  the  Services,  as 
part  of  the  materiel  acquisition  process,  came  under  criticism  by  Congress 
for  Inefficiency.  Reacting  to  this  criticism,  DOD  guidance,  and  the 
results  of  its  own  studies,  the  Army's  current  Operational  Test  and 
Evaluation  (OT&E)  organization  evolved.  This  study  provides  a limited 
evaluation  of  how  well  the  current  Army  OT&E  organization  has  responded 
to  the  criticisms  of  the  past. 

The  history  and  evolution  of  the  Army's  current  OT&E  structure  are 

traced.  The  organization  of  the  US  Army  Operational  Test  and  Evaluation 

Agency  is  discussed.  In  an  analysis,  five  areas  of  major  Congressional 

criticism  are  considered.  The  DOD  guidance  covering  each  area  is  given, 

and  the  Army's  current  policies  and  procedures  which  ameliorate  the 

criticisms  are  cited.  The  Army's  OT&E  policies  are  found  to  be  responsive 

* 

to  history  of  the  recent  past.  Present  OT&E  procedures:  provide  adequate 

evaluation  of  a new  materiel  system  prior  to  a production  commitment; 
respond  to  user  needs;  assist  in  measuring  incremental  effectiveness  of 
new  systems;  and  produce  an  independent  and  objective  evaluation  of  the 
operational  effectiveness  of  new  materiel. 

Some  future  areas  of  interest  are  discussed  which  will  assist  the 
continuation  of  a viable  OT&E  organization  within  the  Army. 
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SECTION  I 
INTRODUCTION 

Purpose  of  the  Study  Project 

In  the  last  five  years  there  have  been  few  publications  on  the  subject 
of  the  Army's  operational  test  and  evaluation  (OT&E)  efforts.  Since  1970, 
the  field  has  seen  significant  changes  as  part  of  the  revamping  of  the 
materiel  acquisition  process  in  the  Department  of  Defense.  These  changes 
have  focused  on  the  important  contributions  of  OT&E  in  times  where  funding 
has  demanded  efficiency  in  systems  acquisition.  The  new  Department  of 
Defense  (DOD)  policies  for  OT&E  are  reflected  in  DOD  Directive  5000.3, 

Test  and  Evaluation.  This  report  traces  the  recent  history  of  the  Army's 
OT&E  organization,  and  investigates  how  well  its  efforts  respond  to  DOD 
guidance  and  past  Congressional  criticism  of  operational  testing.  An 
understanding  of  this  evolution  should  be  of  value  to  those  responsible 
for  the  development,  production,  and  deployment  of  materiel  systems  for 
our  Army.  It  is  hoped  that  the  report  will  motivate  other  studies  in  the 
OT&E  area  which  will  add  to  the  body  of  knowledge  on  operational  testing, 
and  contribute  to  a greater  appreciation  of  OT&E  throughout  the  Army. 

To  this  end,  the  bibliography  has  been  annotated  to  assist  interested 
researchers. 


Goals  of  the  Project 
Specific  goals  for  this  report  are: 


1.  To  research  and  relate  the  historic  rationale  for  OT&E  as  it 
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is  required  presently  by  DOD  and  as  It  applies  to  the  Department  of  the 
Army. 

2.  To  trace  the  path  from  the  major  criticisms  of  OT&E  in  the  materiel 
acquisition  process  to  the  regulations  which  prescribe  its  requirement  in 
the  Army. 

3.  To  examine  the  Army's  organization  for  OT&E. 

4.  To  examine  how  well  the  Army  is  fulfilling  its  major  OT&E  roles. 

5.  To  sample  the  Operational  Test  and  Evaluation  efforts  to  date. 

6.  To  look  into  future  trends  in  the  Army's  OT&E. 


Definitions 


Because  it  provides  a common  denominator,  the  definition  of  operational 

test  and  evaluation  as  stated  in  DOD  Directive  5000.3  is  quoted  below. 

Operational  Test  and  Evaluation  (OT&E).  OT&E  is  that  test 
and  evaluation  conducted  to  estimate  the  prospective  system's 
military  utility,  operational  effectiveness,  and  operational 
suitability  (including  compatibility,  interoperability, 
reliability,  maintainability,  and  logistic  and  training 
requirements),  and  need  for  any  modifications.  In  addition, 

OT&E  provides  information  on  organization,  personnel  require- 
ments, doctrine,  and  tactics.  Also,  it  may  provide  data  to 
support  or  verify  materiel  in  operating  instructions,  publi- 
cations, and  handbooks.  OT&E  will  be  accomplished  by  opera- 
tional and  support  personnel  of  the  type  and  qualifications 
of  those  expected  to  use  and  maintain  the  system  when 
deployed,  and  will  be  conducted  in  as  realistic  as  operational 
environment  as  possible.  OT&E  will  normally  be  conducted  in  . 
phases,  each  keyed  to  an  appropriate  decision  point.  (20:3) 

Operational  testing  (OT)  also  prescribed  to  be  separate  from  develop- 
ment testing  (DT),  which  is  generally  technically  oriented,  e.g.,  toward 


1 This  reference  notation  will  be  used  throughout  the  study.  The  first 
number  refers  to  the  source  as  listed  in  the  bibliography.  The  second 
is  the  page  number  within  the  reference  document.  A single  number 
refers  to  an  interview  source. 
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verification  of  system  specifications  and  technical  performance  parameters. 
Operational  testing  is  different  from  DT  in  that  it  is  to  be  executed  in 
an  operational  environment  and  performed  by  typical  user  troops. 

There  is  a difference  between  an  operational  test  and  the  related 
evaluation.  The  former  is  that  effort  which  actually  causes  the  prospec- 
tive system  to  perform  in  the  field  and  yield  appropriate  data.  The 
latter  is  the  process  of  using  this  data,  analyzing  it  along  with  other 
pertinent  information  such  as  related  studies,  and  evaluating  how  well 
the  system  meets  its  expected  performance.  A simpler  description  is, 
perhaps,  "...  'test'  refers  to  physical  activities  designed  to  secure 
data,  while  'evaluation'  refers  to  the  mental  activity  used  in  processing 
the  test  results  and  other  relevant  information  to  get  useful  conclusions." 
(25:46) 


Scope  and  Limitations 

The  study  effort  consisted  of  research  of  the  rationale  for  OT&E,  the 
corresponding  policies  and  requirements  as  they  exist  in  DOD  today,  and 
a survey  of  Army  OT&E  efforts  on  contemporary  systems.  Guidance  and 
Impressions  of  the  Army's  success  in  its  OT  program  were  obtained  by 
Interviews  with  Department  of  the  Army  and  Department  of  Defense  staff 
personnel.  A non-attribution  guarantee  was  provided  selected  interviewees 
in  an  effort  to  gain  candid  impressions,  either  favorable  or  unfavorable. 

^ For  a history  of  the  difference  between  DT  and  OT  in  the  Army,  the 
reader  is  directed  to  reference  25. 
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Because  of  time  limitations,  it  was  necessary  to  be  selective  in  the 
criteria  used  to  analyze  the  Army's  OT&E  efforts.  DOD  Directive  5000.3, 
which  may  be  thought  of  as  the  "requirements"  document,  contains  many 
statements  of  requirement  or  policy  which  could  be  so  used.  The  major 
ones,  however,  are  those  which  have  in  the  past  been  almost  chronic 
criticisms  from  the  Congress,  or  which  most  significantly  affect  the 
adequacy,  quality,  and  credibility  of  the  DOD's  test  and  evaluation  program. 
Five  criticisms  are  covered  in  four  areas  in  the  analysis  below. 

In  discussing  the  organization  and  roles  of  the  US  Army  Operational 
Test  and  Evaluation  Agency,  there  is  no  detailed  discussion  of  the  general 
management  of  the  operational  testing  of  nonmajor  systems  (Categories  II 
and  III) . Comprehensive  research  of  this  testing  would  have  required  a 
trip  to  Headquarters,  US  Army  Training  and  Doctrine  Command  at  Fort 
Monroe,  VA.  Time  prohibited  such  a trip. 


SECTION  II 

HISTORY  OF  THE  OT&E  REQUIREMENT 

The  Department  of  the  Army  (DA)  has  been  struggling  since  1965  with 
the  establishment  of  an  effective  organization  for  testing  as  part  of 
its  major  weapons  acquisition  system.  There  has  resulted  a series  of 


studies  and  reorganizations  of  T&E  agencies.  Two  of  the  study  efforts, 
the  study  of  Army  Test  and  Evaluation  (SATE)  and  the  DA  Board  of  Inquiry 
on  the  Army  Logistics  System  (Brown  Board) , made  a careful  and  needed 
distinction  between  materiel  testing,  now  known  as  development  testing, 
and  operational  evaluation.  (3:B-1) 

In  spite  of  the  fact  that  the  Army  had  initiated  fruitful  corrective 
measures,  along  with  the  other  Armed  Services  it  suffered  severe  criticism 
from  the  Congress  of  its  acquisition  process.  The  interested  researcher 
can  explore  literally  hundreds  of  readings  on  this  criticism.  One  of 
the  particular  areas  that  Congress  had  been  wont  to  cite  was  the  improper 
or  ill-advised  testing  of  defense  systems.  Key  examples  of  this 
Congressional  tone,  expressed  by  the  House  and  Senate  Armed  Services 
Committees  from  1969  to  1972,  are  cited  topically: 

1.  Operational  testing  was  not  started  early  enough  in  the  acquisition 

cycle.  (29:3)  (28:5) 

2.  Service  must  identify  user  problems  and  insure  that  the  new 
system  responds  to  them.  (31:77) 


3.  System  suitability  must  be  determined  prior  to  commitment  to 
production.  (31:82) 

4.  No  one  in  the  acquisition  process  knows  the  value  of  an  incre- 
mental improvement  in  performance  factors  such  as  rate  of  fire,  vehicle 
speed,  etc.  (31:109,  131) 

5.  Tests  do  not  appear  to  be  independent  from  influences  of  the 

developer;  objectivity  of  the  testing  is  doubtful.  (31:8,  15)  (29:9) 


By  1969,  the  Department  of  Defense  had  already  commenced  to  investigate 
and  correct  the  problem.  Specifically  addressing  the  objectivity  of 
testing  with  the  Army,  The  Blue  Ribbon  Defense  Panel  reported  on  1 July 
1970: 


In  the  current  Army  system,  OT&E  is  subordinate  to 
organizations  which  are  also  responsible  for  development 
of  the  systems  and  equipment  tested.  Both  Engineering 
Tests  and  Service  Tests  are  performed  by  the  Army's  Test 
and  Evaluation  Command  (TECOM)  which  is  a subordinate 
element  of  the  Army  Materiel  Command  (AMC).  (3:20) 

...theoretically  it  is  not  in  the  best  interests  of 
unbiased  and  objective  OT&E  to  have  those  who  perform 
it  report  through  the  developer  to  the  Chief  of  Staff 
level  where  important  decisions  may  rely  extensively 
on  test  results  or  expert  but  basically  subjective 
evaluations.  (3:21) 


On  11  February  1971,  Deputy  Secretary  of  Defense  (DepSecDef)  Packard 

formally  responded  to  the  Blue  Ribbon  Panel's  Reports,  and  issued  a 

memorandum  to  the  Services  initiating  two  critical  actions: 

"...it  is  apparent  to  me  that  this  (OT&E)  can  best  be 

performed  by  an  agency  which  is  separate  and  distinct 

from  the  developing  command  and  which  reports  the 

results  of  its  test  and  evaluatiqn  efforts  directly 

to  the  Chief  of  the  Service.  (18:1)  (Underlining  is  author's.) 
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He  further  called  for  a dedicated  OT&E  office  to  provide  staff 

assistance  directly  to  the  Chief  of  Service,  and  continued: 

"As  a second  step,  I am  establishing  a Deputy  Director 
for  Test  and  Evaluation  within  ODDR&E  with  across  the 
board  responsibilities  for  OSD  in  test  and  evaluation 
matters.  This  office  will  review  and  approve  test  and 
evaluation  plans  prepared  by  the  Service  and  will 
provide  an  assessment  of  results  obtained."  (18:2) 

The  new  office  was  that  of  the  Deputy  Director  of  Defense  Research 
and  Engineering,  Test  and  Evaluation  (DD(T&E)).  The  first  Deputy  Director 
of  the  new  T&E  unit  was  LTG  Alfred  D.  Starbird,  USA  (Ret.).  His  office 
published  the  fundamental  DOD  Directive  (5000.3)  on  Test  and  Evaluation 
on  19  January  1973,  but  even  before  then  had  been  quite  active  in  planning 
and  implementing  test  and  evaluation  guidelines  for  the  Services.  The 
1973  directive  incorporated  and  superceded  a total  of  three  memoranda 
published  by  DepSecDef  in  1971.  (20:1) 

The  reform  efforts  of  DOD  were  not  focused  solely  on  T&E  during  this 
period.  The  above  history  is  only  part  of  this  work  which  brought  sig- 
nificant change  to  the  entire  materiel  acquisition  process  within  DOD. 
Signal  in  the  new  policies  established  was  the  publication  of  DOD 
Directive  5000.1,  Acquisition  of  Major  Defense  Systems,  on  13  July  1971. 

It  laid  the  ground  rules  for  T&E  as  part  of  the  acquisition  cycle  as  well 
as  outlining  virtually  all  other  aspects  of  the  revised  acquisition 
policies  within  DOD. 

Test  and  evaluation  shall  commence  as  early  as 
possible.  A determination  of  operational  suitability, 
including  logistic  support  requirements,  will  be  made 
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prior  to  large-scale  production  commitments,  making 
use  of  the  most  realistic  test  environment  possible 
and  the  best  representation  of  the  future  operational 
system  available.  The  results  of  this  operational 
testing  will  be  evaluated  and  presented  to  the  DSARC 
at  the  time  of  the  production  decision.  (19:5) 

Congress  also  clarified  its  interest  in  OT&E  in  1971  by  adding  Section 
506  to  the  1972  Authorization  Act.  The  section  amended  the  annual  report 
of  the  Secretary  of  Defense  to  require  the  reporting  of  OT&E  results  on 
all  weapon  systems  for  which  procurement  funds  were  being  requested. 

(Public  Law  92-156,  85  Stat.  423,  Nov.  17,  1971.) 

The  Army's  OT&E  effort  during  the  period  1970  to  1972  could  be 
typified  as  one  of  response  to  the  new  policies  and  search  for  improvement 
of  the  areas  in  which  the  Services  had  been  criticized.  While  AMC's  Test 
and  Evaluation  Command  continued  the  expanded  service  tests  which  included 
the  use  of  typical  troops  under  field  conditions,  the  test  plan  now  was 
approved  by  the  Continental  Army  Command  (CONARC) , the  user,  through  its 
Combat  Developments  Command  (CDC) . Unfortunately,  this  system  generally 
delayed  confirmation  of  reliability  and  maintainability  estimates,  as 
well  as  most  examinations  of  organization,  doctrine,  and  tactics,  until 
after  the  prospective  system  was  already  in  production.  Thus,  an  effec- 
tive evaluation  could  only  recommend  modifications  or  indicate  shortcomings 
almost  after  the  fact.  (17:2) 

After  the  DepSecDef  memo  of  11  February  1971,  the  Assistant  Chief  of 
Staff  for  Force  Development  became  the  OT&E  focal  point  for  the  Army  staff. 
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Independent  evaluation  was  attempted  by  making  CDC  responsible  for  OT&E 
or  its  equivalent.  CDC,  however,  was  subordinate  to  CONARC,  the  user. 

Army  Regulation  1000-1  was  published  on  23  June  1972,  describing 
revised  policies  for  materiel  acquisition.  In  keeping  with  these  policies, 
a formal  study  group  under  MG  John  T.  Car ley,  developed  the  plan  for  an 
Army  operational  test  agency  to  be  entirely  independent  of  other  Army 
commands.  The  US  Army  Operational  Test  and  Evaluation  Agency  (OTEA)  was 
formally  activated  on  25  September  1972.  (17:4)  Its  organization  and 

functions  are  prescribed  in  AR  10-4.  (ref.  6) 

In  a 1974  change  to  DOD  Directive  5000.3  "independence"  of  the  Service 
evaluator  was  prescribed:  "...there  will  be  one  major  field  agency 

separate  and  distinct  from  the  developing/procuring  command  and  from  the 
using  command  which  will  be  responsible  for  OT&E...."  (20:3)  (Author's 

underlining.)  This  represented  a change  from  the  initial  DepSecDef  memo 
which  called  only  for  separation  from  the  developing  command.  It  also 
effectively  dictated  that  each  Service's  Independent  evaluation  agency 
could  no  longer  satellite  on  a major  command,  which  the  Army  had  already 
recognized. 

While  the  Army's  new  independent  evaluation  agency  was  transitioning 
into  effective  being,  it  was  the  subject  of  an  April  1974  report  by  the 
Army  Materiel  Acquisition  Review  Committee  (AMARC) . The  AMARC's  recommen- 
dations were  based  on  observations  made  in  a time  when  the  Army  was  still 

• 
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settling  many  testing  Issues  and  responsibilities  following  the  establish- 


ment of  OTEA.  The  AMARC  cited  the  following  major  recommendations  per- 
taining to  operational  testing: 

a.  Present  independent  DT  and  OT  evaluations  at  IPR/ASARC 
meetings. 

b.  Emphasize  difference  between  DT  and  OT,  based  primarily 
on  technical  orientation  of  DT  and  operational  orientation 
of  OT. 

c.  Retain  DT  tasks  in  AMC,  and  OT  tasks  in  TRADOC  and  OTEA. 

d.  Emphasize  independence  of  test  design  and  evaluation, 
rather  than  separate  testing,  as  a key  to  independence  of  OT. 

e.  Place  OTEA  directly  subordinate  to  the  OCSA.  (5:22-23) 

The  above  represents  an  overview  of  the  atmosphere  in  which  the 
current  operational  test  organization  and  philosophy  came  into  being. 
The  next  section  will  highlight  OTEA  and  OT&E  organization  within  the 
Army. 


i 
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SECTION  III 

US  ARMY  CONCEPT  AND  IMPLEMENTATION  OF  OT&E 

The  Army's  policies  for  OT&E  are  sec  forth  in  succinct  style  in 
several  Army  regulations.  AR  1000-1  (ref.  9)  provides  the  relation  of 
operational  testing  to  the  systems  acquisition  process.  AR  70-10  (ref.  7) 

► 

lays  out  the  detailed  implementation  of  DOD  Directive  5000.3.  It  covers 
both  DT  and  OT.  Related  also  are  AR  71-3  (ref.  8),  User  Testing,  and 
AR  10-4  (ref.  6),  OTEA  Organization  and  Functions.  In  the  Army,  "All  0T 
is  the  responsibility  of  and  is  managed  by  the  Operational  Test  and 
Evaluation  Agency  (OTEA)."  (7:1-2)  This  includes  joint  user  testing. 
(6:2) 

AR  70-10  is  the  key  document  for  OT&E.  It  was  only  after  four  major 
staff ings  and  a great  deal  of  coordination  with  DD(T&E)  that  this  regu- 
lation was  published. in  its  current  form.  (ref.  21,  22,  23,  and  48.) 

To  provide  the  reader  an  overview  of  the  relation  of  OT&E  to  the  materiel 
acquisition  process.  Figure  1-1  from  the  regulation  has  been  included. 

In  this  regulation,  the  Army  has  defined  a range  of  operational  tests 
to  satisfy  its  needs.  The  "normal"  OT's  are  I,  II,  and  III,  and  are 
Included  in  the  acquisition  process  as  shown  in  the  figure.  Another  type 
of  operational  test  is  defined  also:  "(Force  Development  Testing  and 

Experimentation)  FDTE  includes  innovative  and  operational  feasibility 
* testing  and  may  support  the  combat  and  force  development  and  the  materiel 

acquisition  processes  by  providing  essential  information  at  decision 

' 
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Figure  1 Materiel  Acquisition  Process  Chart. 
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reviews."  (7:2-9)  Appropriate  exemplary  schedulings  of  FDTE  are  shown 
on  the  figure. 

The  categories  of  Army  systems  which  apply  to  testing  are  given  In 
AR  10-4.  Major  systems  are  as  defined  In  DOD  Directive  5000.1;  however, 
the  Army  has  three  levels  of  nonmajor  systems,  categories  I through  III, 
which  are  classed  in  decreasing  order  of  cost  and  visibility.  (6:1) 

OTEA  manages  the  testing  of  all  systems,  but  directly  executes  only  the 
OT&E  of  major  and  selected  nonmajor  (category  I)  systems  and  major  FDTE 
as  required.  (6:2)  DOD  guidance  in  testing  lower  categories  of  systems 
Is  given  In  DOD  Directive  5000.3.  The  directive  "...provides  principles 
to  be  applied  by  the  DOD  Components  In  their  acquisition  of  Defense 
Systems  that  do  not  fall  in  the  'major  acquisition  programs'  category." 
(20:1)  OTEA  manages  nonmajor  systems  (categories  II  and  III)  under  this 
guidance. 

Other  of  OTEA's  general  functions  include  involvement  In  the  planning, 
conducting,  and  evaluation  of  user  testing  and  formulation  of  user  test 
policy.  Thus,  OTEA  serves  as  the  Army's  focal  point  for  test  programs 
and  information. 


In  performing  its  principal  duties,  OTEA  has  contact  with  the  project 
offices  of  all  major  Army  systems.  It  assists  the  project  manager  in 
formulating  the  required  coordinated  test  program.  This,  as  well  as 
other  coordination  with  the  project  office,  is  accomplished  through  the 
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Test  Integration  Working  Group  as  prescribed  in  Chapter  4 of  DA  Pamphlet 
70-21,  The  Coordinated  Test  Program. 


OTEA's  products  during  a test  cycle  are,  in  order:  an  independent 

evaluation  plan,  a test  design,  a field  test  report,  and  an  Independent 
evaluation  report.  It  is  the  last  of  these  which  is  presented  to  the 
appropriate  decision  review. 

OTEA  coordinates  with  other  agencies  during  the  formulation  of  the 
independent  evaluation  plan  and  the  test  design.  To  properly  test  the 
prospective  system,  OTEA  must  have  the  user's  criteria.  The  user  must 
be  involved  by  providing  scenarios,  doctrine,  tactics,  threat  and  the 
like,  which  impact  on  the  system.  The  T&E  agencies  of  the  developing 
command,  DARCOM,  must  be  coordinated  with  to  assure  that  there  is  no 
unnecessary  duplication  of  testing.  Thus,  although  the  four  documents 
mentioned  above  are  OTEA  products,  there  is  ample  opportunity  for  other 
members  of  the  testing  community  to  contribute  to  the  viability  of  opera- 
tional testing. 

OTEA's  field  test  teams  are  generally  composed  of  a cell  of  at  least 
three  full-time  officers  who  supervise  and  control  the  test  execution 
under  the  guidance  of  the  test  design  plan.  The  entire  test  force  con- 
sists of  personnel  in  the  strengths  and  skills  required  for  the  specific 
test.  An  officer  from  the  test  installation,  normally  of  0-6  rank,  is 
appointed  as  the  Test  Director.  He  also  supervises  test  execution  and 
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signs  the  field  test  report.  By  using  someone  of  appropriate  rank  who  is 
familiar  with  the  installation,  test  support  coordination  problems  are 
minimized.  The  Deputy  Test  Director  of  an  operational  test  is  normally 
the  senior  full-time  OTEA  officer.  He  continues  to  be  rated  by  his  chain 
of  command  within  OTEA.  During  execution,  the  designated  OTEA  Test  Manager 
will  visit  the  test  site  as  the  representative  of  Commander,  OTEA.  The 
Test  Manager  is  specifically  dedicated  at  all  times  to  support  the  test, 
furnish  guidance,  and  assist  in  resolving  problems  as  required. 

The  independent  evaluation  report  is  written  by  the  OTEA  test  evaluator. 
In  the  normal  course  of  his  duty,  he  coordinates  in  the  test  design,  and 
is  involved  in  the  resolution  of  questions  which  may  arise  during  test 
execution.  Along  with  the  field  test  report,  he  uses  related  studies, 
simulations,  and  the  results  of  other  system  tests  to  prepare  the  indepen- 
dent evaluation  for  the  approval  of  the  OTEA  commander,  who  reports 
directly  to  the  Chief  of  Staff  of  the  Army.  The  independent  evaluation 
report  is  normally  briefed  by  OTEA  directly  to  the  DSARC  or  ASARC  for 
major  systems.  (7:3-3) 

The  preceding  several  pages  have  provided  an  encapsulated  version  of 
the  OT&E  structure  for  major  and  selected  nonmajor  (category  I)  systems 
within  the  Army.  The  next  section  is  an  analysis  of  how  that  structure 
serves  to  satisfy  the  guidance  of  DOD  which  are  based  on  the  Congressional 
comments  of  the  early  1970' s. 


SECTION  IV 
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ANALYSIS 

The  current  organization  for  OT&E  is  relatively  new.  Insufficient 
time  has  passed  since  the  establishment  of  OTEA  to  trace  a single  system 
through  its  normal  operational  test  cycles  and  into  deployment.  It  is, 
of  course,  the  future  post-deployment  performance  of  a system  that 
ultimately  provides  an  indication  of  the  efficiency  of  OT&E  concepts. 

Today,  however,  the  Army's  OT&E  organization  can  be  measured  against 
its  intended  purposes  through  the  medium  of  DOD  guidance  and  the 
Congressional  criticism  cited  earlier.  Addressed  below  is  the  question, 
"Do  pertinent  regulations  and  policies  prescribe  compliance  with  DOD 
Directive  5000.3  and  ameliorate  the  areas  of  past  criticism  of  OT&E?" 

In  researching  a response  to  this  issue,  OT&E  documentation  was  sampled. 
The  sampling  provided  an  indication  of  how  the  policies  are  being  carried 
out,  and  an  illustration  of  OT&E's  responsiveness. 

In  the  following  subsections,  the  areas  of  Congressional  criticism 
will  be  summarized  without  reference,  along  with  the  corresponding  policy 
from  DOD  Directive  5000.3.  The  areas  of  Congressional  comment  are  from 
Section  II. 

Testing  Prior  to  Production 

Congress:  Operational  testing  not  started  early  enough  in  acquisition. 

System  suitability  not  checked  before  production  of  weapon. 
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Acquisition  programs  will  be  so  structured  that  at  least 
an  initial  phase  of  operational  test  and  evaluation  (IOT&E) 
will  be  accomplished  prior  to  the  first  major  production 
decision. 

Paragraph  2-3  of  AR  70-10  states  specifically  that  "OT  I and  II  relate 
to  the  initial  phase  of  operational  test  and  evaluation  (IOT&E) .. .prescribed 
in  DOD  Directive  5000.3.  0T  III  may  support  (IOT&E)  when  there  is  a need 
to  the  full  production  decision."  The  regulation  allows  a waiver  on 
approval  at  Department  of  the  Army  level.  Based  on  the  author's  experience 
and  a search  of  OTEA's  documentation,  there  has  been  no  deviation  from 
this  policy  for  a major  or  selected  nonmajor  system. 

Historically,  OT  I and  OT  II  have  served  as  IOT&E.  OT  III  has 
generally  been  associated  with  test  of  an  initial  or  post  production 
model  of  the  system  as  shown  on  figure  1.  The  question  has  been  raised 
to  the  author  on  several  occasions,  however,  "Is  0T  I really  a valuable 
test?"  A reflection  on  the  potential  of  such  a test,  and  in  light  of 
the  regulations  which  prescribe  it,  indicate  a reply  in  the  affirmative. 
While  OT  I may  not  test  an  advanced  prototype  of  a system,  from  an  opera- 
tional view  it  has  the  potential  for  establishing  man-machine  interface 
and  verifying  the  adequacy  of  concepts  connected  with  the  system.  An 
example  is  cited  from  the  forthcoming  XM1  Army  Battle  Tank  OT  I.  The 
crew's  ability  to  quickly  load  the  main  gun  while  on  the  move  will  be 
tested  under  operational  conditions.  This  exercise  will  result  in 
checking  the  adequacy  of  procedures  and  can  provide  useful  information 


related  to  the  Interior  configuration  of  the  tank  and  an  insight  into 
how  the  tank  night  be  combat  loaded  most  efficiently. 

OT  II  provides  the  most  critical  test  for  the  prospective  system. 

This  is  a matter  of  policy  within  OTEA  (47)  and  its  logic  is  sound.  It 
is  OT  II  which  normally  checks  the  system  in  full  scale  development  (FSD) 
prior  to  initial  production.  In  FSD,  the  system  design  becomes  virtually 
fixed,  and  future  change  becomes  difficult.  A favorable  decision  from 
OT  II,  reflected  in  DSARC  III,  commits  the  system  to  the  point  where  a 
delay  or  major  change  may  have  serious  economic  consequences. 

Both  OT  I and  OT  II  serve  to  check  the  system  prior  to  a commitment 
to  production.  They  represent  the  action  taken  by  the  Army  to  insure  a 
"try-before-buy"  concept  is  implemented. 

. Response  to  User  Requirements 

Congress:  Must  identify  user  problems  and  insure  that  system  properly 

responds. 

DOD:  OT&E  is. .. conducted  to  estimate  the. .. system' s military 

utility  (and)  operational  effectiveness....  Insure  that 
OT&E  is  effectively  planned  and  conducted. 

Both  AR  71-3  and  AR  70-10  relate  to  the  identification  of  user  needs 
and  establishing  a system  to  insure  that  a proper  response  is  made  to 
those  needs.  In  describing  the  route  from  need  to  response  paragraph 
3- 2b  of  AR  71-3  on  OT  policy  states,  "There  must  be  a clear  thread  from 


item/system  requirements,  ...to  the  test  design  and  test  criteria  for 
each  phase  of  testing."  The  Coordinated  Test  Program  (CTP)  is  established 
to  do  just  this.  Both  critical  issues  and  the  more  detailed  test  criteria 
are  used,  "...critical  issues  for  testing  will  be  identified  by  the 
materiel  developer,  the  combat  developer,  and  other  participants  in  CTP 
coordination."  (7:2-3)  Test  criteria  are  jointly  defined  by  both  the 
materiel  and  combat  developers,  representing  the  user,  and  coordinated 
with  the  operational  tester  for  insertion  into  the  CTP.  (8:3-3)  Also 

f • 

inserted  into  operational  testing  are  the  user's  concepts  for  organization 
and  doctrine  as  well  as  necessary  training  and  maintenance.  Support 
packages  in  each  of  these  areas  for  use  of  the  operational  tester  are 
required  no  later  than  OT  II.  (7:2-2) 

Evaluation  of  the  system's  response  to  user  needs  then  rests  with 
the  independent  view  presented  at  the  ASARC  and  DSARC  level  for  major 
systems.  "The  (CTP)  will  provide. .. operational  testers  and  evaluators 
with  criteria,  against  which  their  tests  will  be  designed  and  the  data 
evaluated."  (7:1-2)  This  is  further  emphasized  in  paragraph  2-6  of 
AR  70-10  which  states  that  OT  will  provide  data  to  estimate  from  "...the 
user  viewpoint  a system's  desirability,  considering  systems  already 
available  and  the  operational  benefits  or  burdens  associated  with  the 
new  system."  (7:2-7) 

The  above  then  provides  the  policy  under  which  the  OTEA  operates. 

In  surveying  the  operational  test  which  have  been  executed  in  the  recent 
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past,  It  was  observed  that  the  user's  Interests  habitually  were  represented. 
The  test  designs  and  evaluations  are  replete  with  examples  in  which,  for 
example,  the  user's  doctrine  has  been  encompassed,  user  troops  participate, 
and  reliability  and  supportability  are  checked.  All  OTEA  evaluations 
address  the  adequacy  of  the  test  as  a user  test.  One  of  the  by-products 
of  involving  the  user  is  to  sharpen  his  perception  of  how  he  states  his 
requirements.  (47)  Being  involved  in  operational  testing  provides  to 
the  user  a vivid  demonstration  of  what  his  criteria  means  in  terms  of 
realizable  equipment,  performance,  and  doctrine. 

Thus,  not  only  in  OT  policies,  but  in  T&E  execution  the  user's  needs 
are  being  recognized  and  fulfilled  in  the  Army's  program. 


** 


Value  of  Improvements 


Congress:  No  one  in  the  acquisition  process  knows  the  value  of  an 

incremental  improvement  in  performance  factors  such  as  speed. 

DOD:  OT&E  will  estimate  military  utility  and  operational  effective- 

ness. 

The  issue  of  added  value  by  or  to  the  new  system  is  addressed  in  at 
' least  two  different  ways  in  the  Army's  acquisition  process.  The  first 
is  by  conducting  "substantive,  credible  cost  and  operational  effective- 
ness (analyses)  and  trade-off  (analyses)"  as  prescribed  in  AR  71-3  and 
other  regulations.  This  is  not  an  operational  test  function,  but  is 
assigned  to  appropriate  agencies,  depending  on  the  system  being  evaluated. 
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For  major  systems  this  task  would  normally  be  undertaken  by  the  Concepts 
Analysis  Agency  as  prescribed  in  paragraph  l-14c  of  AR  70-1.  These 
studies  would  provide  for  an  evaluation  of  the  benefit  of  Improved  capa- 
bility generally  before  the  full  scale  development  of  a candidate  system. 

The  second  way  in  which  improved  performance  is  addressed  is  in  the 
manner  in  which  the  candidate  is  operationally  tested.  Typical  user 
troops  are  employed  in  the  test  to  insure  that  a net  gain  in  operational 
effectiveness  in  fact  can  be  realized  when  a better  technical  parameter, 
such  as  aircraft  speed  or  weapon  rate  of  fire,  is  built  into  a piece  of 
equipment.  Another  technique  used  to  provide  an  evaluation  of  increased 
performance  is  to  test  multiple  weapon  types  side-by-side  under  like 
conditions.  The  comparative  mode  of  testing  is  standard  in  0T  II  when 
insufficient  data  is  available  on  a current  system  which  is  being  replaced. 
(8:3-1)  For  example,  the  forthcoming  XM1  Tank  test  will  use  the  M60A1 
tank  as  a baseline  system  for  comparison  against  two  prototypes.  The 
tests  for  the  XM204  light  howitzer  and  the  XM198  medium  howitzer  both 
employ  the  same  technique  by  using  the  current  M102  and  M114A1  howitzers 
as  references  during  OT  II.  Evaluation  of  test  results  then  can  yield 
the  value  of  increased  performance  of  a system  under  operational  conditions. 

Test  Influence 

Congress:  Tests  do  not  appear  to  be  independent  from  influence  of  the 

developer;  objectivity  of  the  testing  is  doubtful. 
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Each  Service  will  have  one  major  field  agency,  separate  and 
distinct  from  the  developing/procuring  command  and  from  the 
using  command,  which  will  be  responsible  for  OT&E  and  report 
the  results  of  its  independent  T&E  directly  to  the  Service 
Chief. 


The  issue  of  Independence  and  objectivity  is  perhaps  the  most  basic 
one  in  that  it  directly  reflects  on  the  credibility  of  not  only  a given 
test,  but  the  Service's  entire  program.  Without  credibility  all  other 
efforts  in  testing  ultimately  may  be  unproductive.  As  mentioned  in 
Section  II,  it  was  the  perceived  lack  of  Independence  in  the  Army's  T&E 
organization  in  the  late  1960's  that  contributed  to  the  establishment  of 
an  entirely  separate  OT&E  agency.  An  examination  of  the  Army's  organi- 
zation of  OT&E  for  major  systems  reveals  that  the  Intended  spirit  of 
Independence  is  being  carried  out,  and  objectivity  of  testing  is  retained. 

Complete  separation  of  operational  testing  execution  from  development 
testing  is  a desirable  goal.  It  would  assist  in  erasing  any  perceived 
influences  of  the  developer  on  user  testing.  In  fact,  in  the  initial 
staffing  of  DOD  Directive  5000.3,  a policy  of  total  separation  was 
attempted.  (48)  Because  of  the  economic  implications  of  completely 
separate  testing  on  all  systems,  however,  it  was  not  required  at  all 
points  in  the  acquisition  process.  Totally  independent  evaluations, 
however,  are  required.  The  Army's  policy  which  stems  from  the  DOD  Direc- 
tive is: 


Objective  testing.  ...OT  will  be  independent  of  DT 
and  normally  will  be  conducted  separately  from  DT. 

However,  DT  and  OT  may  be  combined  when  separation  would 
cause  delay  involving  an  unacceptable  military  risk  or 
an  unacceptable  increase  in  acquisition  costs.  ...DT  and 
OT  test  designs  are  prepared  and  test  results  arc  evaluated 
independently,  regardless  of  the  degree  of  combined  testing. 
(7:2-3) 
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In  the  research  for  this  study,  it  was  discovered  that  independence 
is  most  meticulously  sought  at  all  points  in  the  process  of  an  operational 
test  and  its  evaluation.  For  Instance,  in  the  field  the  Deputy  Test 
Director  from  OTEA  works  for,  but  is  not  rated  by,  the  Test  Director  at 
the  test  Installation.  He  continues  to  have  his  efficiency  rating  from 
his  OTEA  supervisors,  even  though  he  may  work  on  a test  for  six  months  in 
the  field.  The  basic  reason  for  this  policy  is  to  avoid  possible  inter- 
pretation of  Influence  from  the  test  director,  who  is  normally  from  a 
developing  or  using  command. 

The  Independent  evaluation  which  OTEA  prepares  is  subject  to  comment 
by  the  Department  of  the  Army  staff.  However,  by  agreement  with  all 
concerned,  that  evaluation  remains  unchanged  in  its  presentation  to  the 
highest  level  decision  body  for  which  it  is  intended.  (47)  OTEA' s 
responsibility  is  directly  to  the  Army  Chief  of  Staff  through  the  Director 
of  the  Army  Staff.  (6:4)  While  it  is  possible  for  the  Army  staff's 
position  on  a given  major  system  to  differ  from  that  indicated  in  the 
OTEA  evaluation,  the  independent  evaluation  would  still  be  made  available 
to  the  DD (T&E) . (47)  Thus,  in  policy  and  in  practice  the  Army  maintains 

Independence  and  objectivity  in  its  operational  test  program. 

The  foregoing  analysis  has  concentrated  on  the  organizational  and 
policy  aspects  of  the  Army's  OT&E  efforts.  Both  of  these  areas  are 
responsive  to  DOD  guidance.  During  the  research  for  this  study  an  effort 
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was  made  to  ascertain  how  well  the  Army  is  perceived  as  accomplishing 
its  independent  test  and  evaluation  on  major  systems.  It  was  found  that 
both  at  the  Department  of  the  Army  staff  level  and  at  OSD  the  Army's 
OT&E  effort  in  general,  and  OTEA  in  particular,  enjoy  a reputation  for 
excellent  quality.  The  operational  test  program  is  coordinated  well  and 
is  responsive  both  to  the  Army  staff  and  DD(T&E).  (48  and  49)  The  overall 
impression  is  that  the  credibility  of  operational  test  and  evaluation  are 
enhanced  by  OTEA's  execution. 
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SECTION  V 

SUMMARY  AND  CONCLUSIONS 

The  history  of  the  Army's  operational  test  and  evaluation  program 
has  been  one  of  response  and  searching  for  the  most  effective  method  of 
supporting  the  materiel  acquisition  process.  Through  DOD  guidance  the 
Congressional  criticisms  and  Service  studies  of  the  1960's  and  early 
1970' s have  resulted  in  the  current  Army  policies  as  well  as  an  indepen- 
dent evaluation  agency  at  the  top  level  of  Army  management. 

Although  OTEA  is  relatively  young,  an  examination  of  its  operation, 
along  with  the  current  Army  policies,  shows  correction  of  the  errors  of 
past  OT&E.  Adequate  testing  and  evaluation  now  are  executed  prior  to  a 
production  decision  on  a prospective  system.  The  user's  needs  are  encom- 
passed as  a part  of  each  of  the  operational  tests  during  system  acquisition. 
Comparative  testirg  and  the  use  of  troops  from  typical  units  have  provided 
valuable  information  with  which  to  assess  the  effectiveness  and  benefits 
derived  from  prospective  systems.  Credibility  of  OT&E  has  been  estab- 
lished by  complete  Independence  of  the  Operational  Test  and  Evaluation 
Agency  from  other  commands  and  its  chain  of  reporting  directly  to  the 
office  of  the  Chief  of  Staff  of  the  Army. 

t 

* 

Thus,  under  DOD  guidance  the  present  Army  program  for  the  operational 
test  and  evaluation  of  major  and  selected  nonmajor  systems  has  answered 

! 

the  criticism  of  the  past.  Its  major  component,  OTEA,  is  organized  as 
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SECTION  VI 


FUTURE  AREAS  OF  INTEREST 

• *•  t 

To  continue  the  favorable  trend  In  operational  testing,  the  Army's 
testing  community  must  remain  aware  of  the  lessons  of  the  past.  Past 
criticisms  could  be  generalized  by  the  word,  "Inefficiency."  Accordingly, 
the  OT&E  organization  must  continue  to  be  alert  to  potential  repetition 
of  any  Inefficiencies  In  Its  policies. 

With  the  recent  emphasis  on  testing,  the  DOD  already  is  Investigating 
possible  overemphasis.  (49)  Overtesting  Is  costly  to  the  project  manager 
In  terms  of  both  time  and  money.  The  effect  of  potential  overtesting  has 
been  felt  by  at  least  one  project  manager.  In  a message  to  the  Army's 
development  command,  DARCOM,  the  Army  staff  has  asked  for  a review  of 
the  14  months  of  continuous  testing  currently  required  of  the  TACFIRE 
Command  and  Control  system.  (11:1)  Operational  testing  is  only  part  of 
that  extended  project  burden,  but  the  situation  Illustrates  the  point 
that  coordination  and  efficiency  in  testing  remain  areas  of  concern. 

In  an  environment  of  tight  budgets  test  planners  are  naturally  pushed 
toward  Increasing  combined  testing,  especially  the  combination  of  DT  and 
OT.  Tet  the  necessity  for  objective  testing  argues  for  separate  testing. 
Maintaining  the  right  balance  between  the  two  extremes  certainly  will  be 
a continuing  area  of  interest  in  operational  test  planning. 


I 

I — 

There  is,  at  present,  no  basis  for  a comprehensive  feedback  of  opera- 
tional testing's  benefits.  Certainly  there  have  been  immediate  benefits 
to  specific  systems  by  virtue  of  the  problems  surfaced.  However,  an 
insufficient  number  of  systems  tested  have  gone  beyond  initial  production, 
so  that  it  is  not  yet  possible  to  draw  general  implications  of  the  long 
term  benefits  of  OT&E.  Thus,  it  will  be  to  the  Army's  advantage  as  major 
systems  are  deployed  to  observe  their  operational  problems  for  trends 
which  might  serve  to  Improve  OT&E  policies. 

Summarizing  the  above,  it  certainly  will  be  to  the  Army's  future 
advantage  to  maintain  vigilance  on  its  testing  programs  to  insure  that 
all  possible  efficiencies  are  realized.  The  Army  has  established  a 
mechanism  for  doing  this,  the  Test  Integration  Working  Group  (TIWG)  (see 
Section  III).  Those  responsible  for  testing  policy  should  review  TIWG 
proceedings  for  trends  to  insure  that  OT&E  plays  its  most  efficient  role 
in  the  materiel  acquisition  process. 


) 
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ABBREVIATIONS  USED 


AMC  - US  Army  Materiel  Command  (now  DARCOM) 

AR  - Army.  Regulation 

ASARC  - Army  Systems  Acquisition  Review  Council 

CDC  - Combat  Developments  Command 

GONARC  - Continental  Army  Command 

CTP  - Coordinated  Test  Program 

DA  - Department  of  the  Army 

DARCOM  - US  Army  Materiel  Development  and  Readiness  Command 

DD(T&E)  - Deputy  Director  of  Defense  Research  and  Engineering,  Test  and 
Evaluation 

DepSecDef  - Deputy  Secretary  of  Defense 
DSARC  - Defense  Systems  Acquisition  Review  Council 
DSMS  - Department  of  Defense  Systems  Management  School 
DT  - Development  Testing 

FDTE  - Force  Development  Testing  and  Experimentation 

IOT&E  ~ Initial  phase  of  Operational  Test  and  Evaluation 

OSD  - Office  of  the  Secretary  of  Defense 

OT  - Operational  Testing 

OT&E  - Operational  Test  and  Evaluation 

OTEA  - US  Army  Operational  Test  and  Evaluation  Agency 

T&E  - Test  and  Evaluation 

TECOM  — US  Army  Test  and  Evaluation  Command 

TIWG  ~ Test  Integration  Working  Group 


BIBLIOGRAPHY 


This  section  Is  divided  into  three  parts.  Part  I contains  references 
from  a variety  of  sources.  Unless  labeled  to  the  contrary,  all  of  these 
publications  were  published  by  the  US  Government  Printing  Office, 
Washington,  DC.  For  the  sake  of  brevity,  that  portion  of  the  individual 
entries  so  indicating  this  fact  has  been  omitted.  Part  II  contains 
publications  of  the  US  Army  Operational  Test  and  Evaluation  Agency,  Falls 
Church,  VA.  The  listed  volumes  were  surveyed  for  the  elements  of  opera- 
tional testing  discussed  in  the  main  body  of  this  report.  Part  III  lists 
interviews  conducted  with  DOD  and  DA  staff  personnel.  Two  of  these  were 
conducted  with  the  understanding  of  non-attribution. 


PART  I 


1.  Bennett,  John  J. , Department  of  Defense  Systems  Acquisition  Management: 

Congressional  Criticism  and  Concern,  published  in  cooperation  with 
the  Defense  Systems  Management  School,  1974.  — Dissertation  support- 
ing author's  work  at  George  Washington  University  is  extremely 
valuable  reference  which  contains  a Congressional  reference  listing 
of  concerns  and  criticism  in  70  different  areas  each  of  which  is 
detailed  by  subjects. 

2.  Blue  Ribbon  Defense  Panel,  Report  to  the  President  and  the  Secretary 

of  Defense  on  the  Department  of  Defense,  1 July  1970  — See  3. 

3.  Blue  Ribbon  Defense  Panel,  Report  to  the  President  and  the  Secretary 

of  Defense  on  the  Department  of  Defense,  Appendix  F,  Staff  Report 
on  Operational  Test  and  Evaluation.  1 July  1970.  This  is  a 
comprehensive  study  of  OT&E  within  the  Department  of  Defense.  It 
was  used  as  a reference  in  many  Congressional  hearings  in  the 
early  1970’ s. 

4.  Department  of  the  Air  Force,  Air  Force  Test  and  Evaluation  Center, 

Kirtland  AFB,  NM,  Fact  Sheet,  Managing  Air  Force  Test  and  Evaluation, 
AFTEC  Information  Office,  July  1975.  — Gives  history,  mission,  and 
organization  of  US  Air  Force's  OT&E  agency. 

5.  Department  of  the  Army,  Office  of  the  Secretary  of  the  Army,  Report 

of  the  Army  Materiel  Acquisition  Review  Committee  (AMARC) , two 
volumes,  1 April  1974.  This  set  is  a well  written,  broad  coverage 
of  the  Army's  entire  materiel  acquisition  arena. 


30 


» AR  10-4,  Organization  and  Functions.  US  Arm 

Operational  Test  and  Evaluation  Agency.  1 February  1975.  — Details 
OTEA's  mission  and  principal  functions;  provides  definitions  of 
categories  of  nonmajor  systems. 

, AR  70-10,  Research  and  Development.  Test  and 

Evaluation  During  Development  and  Acquisition  of  Materiel. 

1 January  1976.  — Provides  the  concepts,  responsibilities,  policies, 
and  procedures  for  the  Army's  implementation  of  DOD  Dir  5000.3. 

Is  an  excellent  basic  reference  for  research  on  Army  testing. 

_»  AR  71-3,  Force  Development.  User  Testing. 

1 May  1975.  — Gives  policies,  responsibilities,  and  procedures 
for  OTEA's  testing  areas.  Excellent  reference  for  first  level 
breakdown  of  testing  procedures  and  definitions. 

, AR  1000-1,  Basic  Policies  for  Systems 

Acquisition  by  the  Department  of  the  Army.  1 January  1975.  — Provides 
Army's  system  acquisition  policies;  shows  OT&E  relation  with 
acquisition. 

, Office  of  the  Assistant  Chief  of  Staff  for 

Force  Development,  Memorandum  for:  LTG  Williams,  Subject:  OTEA 

Briefing  for  General  Abrams.  19  September  1972,  20  September  1972. 

— One  page  memo  indicates  some  of  Army  staff's  reactions  to  briefing 
on  formation  of  OTEA. 

» DAMA-CSC,  Message,  Subject:  TACFIRE  Testing. 

311445Z  March  1976.  — Requests  DARC0M  to  initiate  detailed  review 
to  eliminate  any  unnecessary  testing. 

> US  Army  Training  and  Doctrine  Command,  ATCD-TM, 

Message,  Subject:  Operational  Test  (OT)  Conditions,  291447Z  August 

1975.  — Promulgates  corrective  action  based  on  improper  portion  of 
an  OT  III. 


> US  Army  Operational  Test  and  Evaluation  Agency, 

Falls  Church,  VA,  DACS-TEZ-A,  Memorandum  to:  Vice  Chief  of  Staff. 

US  Army,  Subject:  OTEA  Board  of  Visitors.  23  July  1975.  --Provides 
synopsis  of  actions  resulting  from  DA  Staff  Board  of  Visitors 
findings  during  visit  to  OTEA  in  February  1975. 

> Pamphlet  containing  OTEA  orientation  infor- 
mation, October  1975.  — Gives  general  data  concerning  mission, 
functions,  and  activities  of  OTEA. 


31 


I 


15. , OTEA  Memo  10-1.  Subject:  Organization. 

Mission,  and  Functions  Manual,  1 October  1975.  — Provides 
responsibilities  to  branch  level  within  OTEA  organization. 

, DACS-TEO-P,  Letter,  Subject:  Exit  Conference 

on  GAO  Survey  of  POD  Test  Organizations  and  Joint  Test  Operations, 
GAO  Code  951202,  5 February  1976.  — Two  page  letter  provides 
general  comments  on  GAO  survey  conducted  November  1975  to  January 
1976. 

17.  , Fact  Sheet,  Subject:  Evolution  of  the  US 

Army  Operational  Test  and  Evaluation  Agency,  undated.  — Five  page 
narrative  traces  establishment  of  OTEA  through  its  organizational 
predecessors  in  the  period  1969  to  1972. 

18.  Department  of  Defense,  Deputy  Secretary  of  Defense  Multi-addressee 

Memorandum,  Conduct  of  Operational  Test  and  Evaluation, 

February  11,  1971. 

19.  Department  of  Defense,  POD  Directive  5000.1,  Subject:  Acquisition 

of  Major  Defense  Systems,  July  13,  1971.  — Later  edition  dated 
December  22,  1975  is  current. 

20.  , POD  Directive  5000.3,  Subject:  Test  and 

Evaluation.  January  19,  1973.  — Changes  have  been  made  through 
May  20,  1975.  Contains  basic  DOD  guidance  in  both  operational 
and  development  testing. 

21.  , Office  of  DDR&E,  Memorandum  for  (Multiple 

addressees  on  Department  of  the  Army  Staff),  Subject:  Draft 

Army  Regulation  No.  70-10,  "Test  and  Evaluation  during  Development 
and  Acquisition  of  Materiel,  ...comments  on,"  LTG  Starbird,  DD(T&E), 
15  June  1973. 

22.  , Memorandum  for  Mr.  John  P.  Tyler,  T&E  Branch, 

OCRD,  Subject:  (same  as  preceding  reference),  RADM  F.  S.  Petersen, 

USN,  Assistant  Director  (Strategic  and  Support  Systems)  Test  & 
Evaluation,  7 September  1973. 

, Memorandum  for  BGEN  John  H.  Boyes,  Subject: 

(same  as  preceding  reference),  LTG  Starbird,  DD(T&E),  3 December 
1973. 

24.  Orr,  LTC  James  M. , US  Army,  The  Effect  of  Operational  Test  Requirements 
on  Army  Project  Management.  Defense  Systems  Management  School, 

Fort  Belvoir,  VA  22C60,  May  1975.  — This  is  an  individual  study 
project  report  by  a member  of  DSMS  Program  Management  Course  75-1. 


32 


: 


I 


25.  Petrolino,  MAJ  Joseph  A. , Jr.,  US  Army,  The  Impact  of  the  AMARC 

Report  on  Army  Testing,  Defense  Systems  Management  Schoo3^-  Fort 
Belvoir,  VA  22060,  October  1975.  — This  Is  an  Individual  study 
project  report  by  a member  of  DSMS  Program  Management  Course  15-2. 

26.  Robbins,  LTC  Clyde  R. , USAF,  "Operational  Test  and  Evaluation,  a 

Look  into  the  Fundamentals,"  Air  University  Review,  26:44-53, 
November  - December  1974.  — Contains  a eonceptuairview  of  OT&E 
in  the  Air  Force. 

27.  US  Congress.  House  of  Representatives.  Committee  on  Government 

Operations.  Government  Procurement  and  Contracting  (Part  4) . 
Hearings  before  a subcommittee  of  the  House  Committee  on  Government 
Operations  on  H.  R.  474,  91st  Cong.,  1st  sess-.-,- -1969. -^This  and 
the  next  six  references  contain  good  examples  of  Congressional 
criticism  of  the  DOD  Materiel  Acquisition  Process. 

28.  . Committee  on  Armed  Services.  Review  of  Army 

Tank  Program.  Report  of  the  Armed  Services  Investigating  

Subcommittee  of  the  House  Armed  Services  Committee.'  Committee 
Print,  1969. 

29.  . Committee  on  Armed  Services.  Army  Procurement 

of  the  M561,  Gama  Goat.  Report  of  the  Armed  Services  Investigating 
Subcommittee  of  the  House  Committee  on  Armed  Services.  Committee 
Print,  1972.  “ ' 

30.  . Senate.  Committee  on  Armed  Services. 

Advanced  Prototype.  Hearings  before  the  Senate  Committee  on  Armed 
Services,  92d  Cong.,  1st  sess.,  1971.' 

31.  . Weapon  Systems  Acquisition  Process.  Hearings 

before  the  Committee  on  Armed  Services,  92d  Congr;— 1st  sess.  , 1971. 

32.  . Committee  on  Government  Operations. 

Acquisition  of  Major  Systems.  Extract  from  Report  of  the  Commission 
on  Government  Procurement,  Volume  2,  December  1972,  Eart  C. , 
Subcommittee  on  Federal  Spending  Practices,  Efficiency,  and  Open 
Government,  1975.  — Although  not  as  current  as  indicated  by  the 
1975  date,  pages  118  through  129  contain  a concise  Congressional 
view  of  OT&E  shortcomings.  The  footnotes  contain  other  references 
for  exploration. 


I 


33 


PART  II 

TEST  DESIGN  PLANS 


Subject 

33.  M48  Product  Improvement  (M48A5)  Verification  Test. 

34.  All  Weather  Short  Range  Air  Defense  System  Cooper- 
ative Operational  Test  (SHORADS  COT),  as  amended. 

35.  Stinger  Man-Portable  Air  Defense  System,  OT  II. 

36.  Utility  Tactical  Transport  Aircraft  System  (UTTAS), 
OT  II. 

37.  New  Army  Battle  Tank,  XM1,  OT  I,  as  amended. 

38.  Baseline  Armor  Reliability  Test  (M48A5/M60A1  BART). 


Date 


21 

May 

75 

19 

May 

75 

24 

Nov 

75 

Nov 

75 

Jan 

76 

Mar 

76 

TEST  REPORT 

39.  XM204,  105mm,  Towed,  Light  Howitzer,  OT  II.  Aug  75 

INDEPENDENT  EVALUATIONS1 


40.  Improved  COBRA  Armament  Program  (ICAP),  Report:  OT-012-ED.  Jan  74 

41.  DRAGON  Training  Equipment,  Report:  OT-095-ED.  Mar  74 

42.  M60A2  Tank,  Report:  OT-006-TEE.  Jul  74 

43.  Nonnuclear  LANCE,  XM251  Warhead,  Report:  OT-003-TEE.  Nov  74 

44.  COBRA/TOW,  Report:  0T-131-TEE.  Mar  75 

45.  Guided  Missile  System,  Surface  Attack,  M47,  May  75 

DRAGON,  Report:  OT- 024-TEE. 

46.  Abbreviated  Independent  Evaluation  of  OT  II  for  6 Nov  75 

XM204  Howitzer,  unclassified  letter. 


All  Independent  evaluations  surveyed,  except  number  46,  are  classified 
Confidential. 


PART  III 


INTERVIEWS 


47.  With  Mr.  W.  Hollis,  Scientific  Advisor,  USAOTEA,  28  January  1976. 

48.  With  DA  Staff  User  Test  officer,  17  March  1976. 

49.  With  OSD  Staff  T&E  officer,  17  March  1976. 

50.  With  LTC  R.  Sugg  and  LTC  W.  Hanson,  Plans  & Operations  Division, 
USAOTEA,  during  period  28  January  1976  to  19  April  1976. 


35 


